Decision Curve Analysis for Personalized Treatment Choice between Multiple Options.

BACKGROUND Decision curve analysis can be used to determine whether a personalized model for treatment benefit would lead to better clinical decisions. Decision curve analysis methods have been described to estimate treatment benefit using data from a single randomized controlled trial. OBJECTIVES Our main objective is to extend the decision curve analysis methodology to the scenario in which several treatment options exist and evidence about their effects comes from a set of trials, synthesized using network meta-analysis (NMA). METHODS We describe the steps needed to estimate the net benefit of a prediction model using evidence from studies synthesized in an NMA. We show how to compare personalized versus one-size-fit-all treatment decision-making strategies, such as "treat none" or "treat all patients with a specific treatment" strategies. First, threshold values for each included treatment need to be defined (i.e., the minimum risk difference compared with control that renders a treatment worth taking). The net benefit per strategy can then be plotted for a plausible range of threshold values to reveal the most clinically useful strategy. We applied our methodology to an NMA prediction model for relapsing-remitting multiple sclerosis, which can be used to choose between natalizumab, dimethyl fumarate, glatiramer acetate, and placebo. RESULTS We illustrated the extended decision curve analysis methodology using several threshold value combinations for each available treatment. For the examined threshold values, the "treat patients according to the prediction model" strategy performs either better than or close to the one-size-fit-all treatment strategies. However, even small differences may be important in clinical decision making. As the advantage of the personalized model was not consistent across all thresholds, improving the existing model (by including, for example, predictors that will increase discrimination) is needed before advocating its clinical usefulness. CONCLUSIONS This novel extension of decision curve analysis can be applied to NMA-based prediction models to evaluate their use to aid treatment decision making. HIGHLIGHTS Decision curve analysis is extended into a (network) meta-analysis framework.Personalized models predicting treatment benefit are evaluated when several treatment options are available and evidence about their effects comes from a set of trials.Detailed steps to compare personalized versus one-size-fit-all treatment decision-making strategies are outlined.This extension of decision curve analysis can be applied to (network) meta-analysis-based prediction models to evaluate their use to aid treatment decision making

Estimating Patient-Specific Relative Benefit of Adding Biologics to Conventional Rheumatoid Arthritis Treatment: An Individual Participant Data Meta-Analysis.

IMPORTANCE Current evidence remains ambiguous regarding whether biologics should be added to conventional treatment of rheumatoid arthritis for specific patients, which may cause potential overuse or treatment delay. OBJECTIVES To estimate the benefit of adding biologics to conventional antirheumatic drugs for the treatment of rheumatoid arthritis given baseline characteristics. DATA SOURCES Cochrane CENTRAL, Scopus, MEDLINE, and the World Health Organization International Clinical Trials Registry Platform were searched for articles published from database inception to March 2, 2022. STUDY SELECTION Randomized clinical trials comparing certolizumab plus conventional antirheumatic drugs with placebo plus conventional drugs were selected. DATA EXTRACTION AND SYNTHESIS Individual participant data of the prespecified outcomes and covariates were acquired from the Vivli database. A 2-stage model was fitted to estimate patient-specific relative outcomes of adding certolizumab vs conventional drugs only. Stage 1 was a penalized logistic regression model to estimate the baseline expected probability of the outcome regardless of treatment using baseline characteristics. Stage 2 was a bayesian individual participant data meta-regression model to estimate the relative outcomes for a particular baseline expected probability. Patient-specific results were displayed interactively on an application based on a 2-stage model. MAIN OUTCOMES AND MEASURES The primary outcome was low disease activity or remission at 3 months, defined by 3 disease activity indexes (ie, Disease Activity Score based on the evaluation of 28 joints, Clinical Disease Activity Index, or Simplified Disease Activity Index). RESULTS Individual participant data were obtained from 3790 patients (2996 female [79.1%] and 794 male [20.9%]; mean [SD] age, 52.7 [12.3] years) from 5 large randomized clinical trials for moderate to high activity rheumatoid arthritis with usable data for 22 prespecified baseline covariates. Overall, adding certolizumab was associated with a higher probability of reaching low disease activity. The odds ratio for patients with an average baseline expected probability of the outcome was 6.31 (95% credible interval, 2.22-15.25). However, the benefits differed in patients with different baseline characteristics. For example, the estimated risk difference was smaller than 10% for patients with either low or high baseline expected probability. CONCLUSIONS AND RELEVANCE In this individual participant data meta-analysis, adding certolizumab was associated with more effectiveness for rheumatoid arthritis in general. However, the benefit was uncertain for patients with low or high baseline expected probability, for whom other evaluations were necessary. The interactive application displaying individual estimates may help with treatment selection

Predicting the treatment response of certolizumab for individual adult patients with rheumatoid arthritis: protocol for an individual participant data meta-analysis.

BACKGROUND A model that can predict treatment response for a patient with specific baseline characteristics would help decision-making in personalized medicine. The aim of the study is to develop such a model in the treatment of rheumatoid arthritis (RA) patients who receive certolizumab (CTZ) plus methotrexate (MTX) therapy, using individual participant data meta-analysis (IPD-MA). METHODS We will search Cochrane CENTRAL, PubMed, and Scopus as well as clinical trial registries, drug regulatory agency reports, and the pharmaceutical company websites from their inception onwards to obtain randomized controlled trials (RCTs) investigating CTZ plus MTX compared with MTX alone in treating RA. We will request the individual-level data of these trials from an independent platform (http://vivli.org). The primary outcome is efficacy defined as achieving either remission (based on ACR-EULAR Boolean or index-based remission definition) or low disease activity (based on either of the validated composite disease activity measures). The secondary outcomes include ACR50 (50% improvement based on ACR core set variables) and adverse events. We will use a two-stage approach to develop the prediction model. First, we will construct a risk model for the outcomes via logistic regression to estimate the baseline risk scores. We will include baseline demographic, clinical, and biochemical features as covariates for this model. Next, we will develop a meta-regression model for treatment effects, in which the stage 1 risk score will be used both as a prognostic factor and as an effect modifier. We will calculate the probability of having the outcome for a new patient based on the model, which will allow estimation of the absolute and relative treatment effect. We will use R for our analyses, except for the second stage which will be performed in a Bayesian setting using R2Jags. DISCUSSION This is a study protocol for developing a model to predict treatment response for RA patients receiving CTZ plus MTX in comparison with MTX alone, using a two-stage approach based on IPD-MA. The study will use a new modeling approach, which aims at retaining the statistical power. The model may help clinicians individualize treatment for particular patients. SYSTEMATIC REVIEW REGISTRATION PROSPERO registration number pending (ID#157595)

A two-stage prediction model for heterogeneous effects of many treatment options : application to drugs for Multiple Sclerosis

Treatment effects vary across different patients and estimation of this variability is important for clinical decisions. The aim is to develop a model to estimate the benefit of alternative treatment options for individual patients. Hence, we developed a two-stage prediction model for heterogeneous treatment effects, by combining prognosis research and network meta-analysis methods when individual patient data is available. In a first stage, we develop a prognostic model and we predict the baseline risk of the outcome. In the second stage, we use this baseline risk score from the first stage as a single prognostic factor and effect modifier in a network meta-regression model. We apply the approach to a network meta-analysis of three randomized clinical trials comparing the relapse rate in Natalizumab, Glatiramer Acetate and Dimethyl Fumarate including 3590 patients diagnosed with relapsing-remitting multiple sclerosis. We find that the baseline risk score modifies the relative and absolute treatment effects. Several patient characteristics such as age and disability status impact on the baseline risk of relapse, and this in turn moderates the benefit that may be expected for each of the treatments. For high-risk patients, the treatment that minimizes the risk to relapse in two years is Natalizumab, whereas for low-risk patients Dimethyl Fumarate Fumarate might be a better option. Our approach can be easily extended to all outcomes of interest and has the potential to inform a personalised treatment approach

Real-world predictions of treatment effects using evidence synthesis for health technology assessment

Background Prognostication and individualized predictions play important roles in clinical decision-making. In clinical practice, prognostic models foresee an individual’s future health condition, whereas prediction models identify subgroups of patients who benefit from a treatment. Aiming to predict individualized treatment effects, risk modelling is a successful method in several clinical areas. Risk modelling consists of two stages. In the first stage, the baseline risk of the outcome is estimated via a prognostic model; and in the second stage, the baseline risk is used as the only effect modifier for the treatment effect estimation. However, risk modelling is limited in cases where treatments are directly compared in single or several randomized clinical trials (RCTs). For most clinical conditions today, since treatment options are numerous, direct comparisons in single mega-trials are impossible. Network meta-analysis (NMA)—a key tool—synthesizes evidence from several studies and compares treatment effectiveness with direct and indirect estimations. By extending the risk modelling approach into an NMA framework, it supports clinical decision-making, allowing for individualized predictions for all available treatment options in a clinical area. In addition, a framework combining several data sources, such as real-world data with RCTs, and aggregate data (AD) with individual participant data (IPD), allows for predictions among RCT and real-world populations. The latter also uses all relevant information, leading potentially to more precise results. Many clinical areas, such as multiple sclerosis (MS), lack established individualized prediction models, which could support patients and their physicians when selecting optimal treatment. Finally, prediction models need evaluation for accuracy and—more notably—clinical usefulness. Decision curve analysis (DCA) is a method recommended for evaluating clinical usefulness of prediction models; however, existing DCA methods only evaluate models developed via single RCT. Established methods for evaluating prediction models from pairwise or NMA are not yet developed. Aim I fill several aforementioned methodological gaps by allowing for individualized predictions in an NMA framework with illustrative examples from the MS clinical area. I also support clinical decision-making processes in the MS clinical area, where many gaps in prognosis and prediction of heterogeneous treatment effects exist. To do so, my objectives include 1. building a prognostic model for predicting relapses among patients with relapsing-remitting multiple sclerosis (RRMS) with observational study data—a data source depicting best real-world conditions; 2. developing a methodological framework by combining prognostic modelling and NMA for individualized predictions under several treatment options; 3. extending the former developed framework by combining several data sources: real-world data with RCTs for predictions in RCT among real-world populations and AD with IPD for using all the relevant information available; and 4. developing methods for evaluating clinical utility of prediction models by extending existing DCA methods. Methods Since there was no established prognostic model, I use high-level statistical methodology to develop a prognostic model for MS. Then, I combined ideas from prognosis research and NMA, extended the risk modelling approach, and built a framework for individualized predictions when several treatment options were available. I further extended this methodology into a general framework where several data sources were combined. Finally, I integrated ideas from DCA and NMA to develop a model which outweighs harms and benefits of each treatment to evaluate prediction model clinical utility. I applied all developed methods on datasets of patients diagnosed with RRMS. I developed a prognostic model identified via the literature and used eight baseline variables, which contribute to estimating the baseline risk score and—in turn—identifying how treatment effects varied across patients. Variables were age, sex, months since last relapse, prior MS treatment, number of prior relapses, expanded disability status scale, number of gadolinium enhanced lesions, and diagnosis years. I used three RCT with IPD and two RCT with AD with evidence about placebo and three active treatments: dimethyl fumarate, glatiramer acetate, and natalizumab. I used placebo arms from nine RCT with IPD and the Swiss Multiple Sclerosis Cohort— a high-quality observational study. Results My developed framework for individualized predictions of treatment effects showed baseline risk plays an important role in estimated treatment effects hence also for optimal treatment recommendations. Whereas natalizumab is considered—on average—the optimal treatment option to minimize the risk of relapsing within the next two years, dimethyl fumarate is the optimal choice for patients with low baseline risk among RCT and real-world populations. When I applied the developed DCA methodology to evaluate clinical utility of one of the MS prediction models, the developed prediction model performs either close or better than other default strategies. Conclusions As a contribution to personalized medicine, my suggested frameworks can be used to make individualized predictions for all available treatments for any clinical condition. In addition, my proposed evaluation method evaluates the clinical utility of such models and potentially provides support to the MS clinical area, which lacks decision-making tools

Ανάπτυξη και επαλήθευση προγνωστικών μοντέλων για δίτιμες εκβάσεις

Σκοπός της παρούσας διπλωματικής ήταν να πραγματοποιηθεί μια ανασκόπηση και αναλυτική περιγραφή των μεθόδων που χρησιμοποιούνται για την ανάπτυξη και επαλήθευση των προγνωστικών μοντέλων σε δίτιμες εκβάσεις, να αναπτυχθεί μία ανασκόπηση της μεθόδου με την οποία μπορεί να κατασκευαστεί μία κλίμακα από ένα τέτοιο προγνωστικό μοντέλο ούτως ώστε να διευκολυνθεί η χρήση του στην καθημερινή πρακτική, καθώς και να πραγματοποιηθεί μία εφαρμογή σε πραγματικά δεδομένα. Τα βήματα που χρειάζονται απαραίτητα για την διαμόρφωση ενός προγνωστικού μοντέλου είναι αρχικά, ο καθορισμός της έκβασης και όλων των υποψήφιων προγνωστικών παραγόντων, και η επιλογή του σωστού τύπου μοντέλου ανάλογα με τα δεδομένα. Συγκεκριμένα για δίτιμες εκβάσεις, επιλέγεται το λογαριθμιστικό μοντέλο παλινδρόμησης. Έπειτα, πρέπει να γίνει μία σωστή διαχείριση ελλειπουσών τιμών ανάλογα με το είδος τους. Το ακόλουθο βήμα είναι η κωδικοποίηση των συνεχών μεταβλητών η οποία γίνεται με πολλές μεθόδους ανάλογα με την μεταβλητή (Διχοτόμηση, Κατηγοριοποίηση, Χρήση της μεταβλητής ως γραμμική, Πολυωνυμική μορφή, Κλασματική πολυωνυμική μορφή και Splines). Ακολούθως, επιλέγεται η μέθοδος επιλογής προγνωστικών παραγόντων κινδύνου (π.χ. βηματική ή bootstrap). Ακόμη, μπορεί να επιλεχθεί ως τελικό μοντέλο το μοντέλο με όλους τους πιθανούς προγνωστικούς παράγοντες κινδύνου, χωρίς περαιτέρω μείωσή τους, μέσω διαδικασίας επιλογής. Εφόσον έχει επιλεχθεί η μέθοδος επιλογής των μεταβλητών του τελικού μοντέλου, ακολουθεί η εκτίμηση των συντελεστών παλινδρόμησης, η οποία συνήθως γίνεται με την μέθοδο μέγιστης πιθανοφάνειας. Παρόλα αυτά νέες μέθοδοι αναφέρονται στην βιβλιογραφία (Μέθοδος ομοιόμορφης συρρίκνωσης, Ποινικοποιημένη εκτίμηση μέγιστης πιθανοφάνειας, μέθοδος LASSO, Μέθοδος κορυφογραμμής), με σκοπό να περιορίσουν την υπερπροσαρμογή του μοντέλου. Απαραίτητη είναι η επικύρωση του τελικού μοντέλου με εσωτερική ή εξωτερική επικύρωση και η αξιολόγηση του ως προς την ικανότητα βαθμονόμησής του, την ικανότητα διαχωριστικότητας του και την κλινική του χρησιμότητα. Τέλος, η δημιουργία κλίμακας βαθμονόμησης για το μοντέλο συμβάλλει στην διευκόλυνση της χρήσης του από τους ειδικούς. Για την εφαρμογή σε δεδομένα, χρησιμοποιήθηκαν δεδομένα από το Γενικό Νοσοκομείο Αθηνών «Λαϊκό» προκειμένου να αξιολογηθούν οι παράγοντες κινδύνου για ύπαρξη του Χρυσίζοντος σταφυλόκοκκου ανθεκτικού στην μεθικιλλίνη (MRSA) κατά την εισαγωγή των ασθενών στο νοσοκομείο. Ένα προγνωστικό εργαλείο για την ταυτοποίηση των ασθενών αυτών θα ήταν χρήσιμο για την απλοποίηση της πολιτικής του ελέγχου του MRSA στα νοσοκομεία. Στην εφαρμογή μετά από προδρομική βηματική μέθοδο επιλογής παραγόντων σε επίπεδο στατιστικής σημαντικότητας 5%, προέκυψαν επτά στατιστικά σημαντικοί παράγοντες κινδύνου για την μόλυνση του σταφυλόκοκκου ανθεκτικού στην μεθικιλλίνη (MRSA): διαβήτης, διαμονή σε γηροκομείο, πρόσφατη χρήση αντιβιοτικών, άνοια / ψυχιατρικές ασθένειες, χρόνια δερματική ασθένεια, πρόσφατη παραμονή σε νοσοκομείο, φύλο. Μετά από εσωτερική επικύρωση στο μοντέλο, προέκυψε τιμή καλής προσαρμογής c-statistic ίση με 1.031 με 95% Δ.Ε (0.891, 1,171), τιμή AUC ίση με 0.768 με 95% Δ.Ε (0.697,0.839) και τιμή διαχωριστικότητας ίση με 0.051. Το προγνωστικό μοντέλο που δημιουργήθηκε, με βάση την εσωτερική επικύρωση που εφαρμόστηκε, φαίνεται ότι μπορεί να προβλέψει ικανοποιητικά την έκβαση. Παρόλα αυτά η εξωτερική επικύρωση θα παρείχε περισσότερες πληροφορίες για την δύναμη του προγνωστικού αυτού μοντέλου. Η μοντέρνα ιατρική βασίζεται, όλο και περισσότερο στα διαγνωστικά, προγνωστικά μοντέλα με σκοπό να ενημερώσει τα άτομα και τους επαγγελματίες υγείας για τους κινδύνους παρουσίας ή εμφάνισης μίας έκβασης και να καθοδηγήσει τις κλινικές αποφάσεις που σκοπό έχουν να μετριάσουν τους κινδύνους αυτούς. Ως εκ τούτου, τα προγνωστικά μοντέλα έχουν όλο και μεγαλύτερη χρησιμότητα στην ιατρική πράξη.This thesis aims to provide a review and a detailed description of the methods used to develop and validate prognostic models for binary outcomes; to develop a review of the method by which a risk score can be constructed from such a prognostic model, to facilitate its use in everyday practice as well as to apply these methods on real world data. Determining the outcome and all candidate prognostic factors, while choosing the right model according to the data are the initial steps required to form a prognostic model. The logistic regression model is appropriate for binary outcomes. As a next step, a proper management of missing values according to their type is necessary. The coding of continuous variables comes next; it can be done in various methods depending on the variable (Dichotomization, Categorization, Linear, Polynomials, Fractional polynomials and Splines). The method of selecting prognostic risk factors is then selected (e.g. stepwise methods or bootstrap). In addition, the final model could include all possible risk factors, without any further reduction, through a selection procedure. Once the model selection method is chosen, the regression coefficients are evaluated, usually through the Maximum Likelihood Method. However, new methods are reported in the literature (Uniform Shrinkage Method, Penalized ML Method, LASSO Method, Ridge Method) to limit model overestimation. It is necessary to validate the final model with internal or external validation and to assess its calibration ability, its descrimination ability, as well as its clinical utility. Finally, creating a risk score for the model contributes to facilitating its use in clinical practice. These methods are applied on data from the Athenian General Hospital “Laiko” in order to assess the risk factors for MRSA colonization on admission of patients. A prognostic tool for identifying these patients would be useful to simplify the MRSA control policy in hospitals. The application of a forward stepwise selection identified seven statistically significant risk factors for MRSA infection: diabetes, nursing home resident, recent use of antibiotics, dementia / psychological disorders, chronic skin disease, recent hospitalization, gender. Consiquently, internal validation in the model were obtained: c-statistic equal to 1.031 with 95% CI (0.891, 1.171), AUC equal to 0.768 with 95% CI (0.697.0.839) and discrimination slope equal to 0.051. The final prognostic model based on the internal validation applied, appears to satisfactorily predict the outcome. However, external validation would provide more information about the validity of this prognostic model. Modern medicine is increasingly based on diagnostic prognostic models to inform individuals and health professionals about the risks of occurrence of an outcome and to guide clinical decisions designed to mitigate these risks. Therefore, prognostic models are becoming more and more useful in medical practice

Development, validation and clinical usefulness of a prognostic model for relapse in relapsing-remitting multiple sclerosis.

BACKGROUND Prognosis for the occurrence of relapses in individuals with relapsing-remitting multiple sclerosis (RRMS), the most common subtype of multiple sclerosis (MS), could support individualized decisions and disease management and could be helpful for efficiently selecting patients for future randomized clinical trials. There are only three previously published prognostic models on this, all of them with important methodological shortcomings. OBJECTIVES We aim to present the development, internal validation, and evaluation of the potential clinical benefit of a prognostic model for relapses for individuals with RRMS using real-world data. METHODS We followed seven steps to develop and validate the prognostic model: (1) selection of prognostic factors via a review of the literature, (2) development of a generalized linear mixed-effects model in a Bayesian framework, (3) examination of sample size efficiency, (4) shrinkage of the coefficients, (5) dealing with missing data using multiple imputations, (6) internal validation of the model. Finally, we evaluated the potential clinical benefit of the developed prognostic model using decision curve analysis. For the development and the validation of our prognostic model, we followed the TRIPOD statement. RESULTS We selected eight baseline prognostic factors: age, sex, prior MS treatment, months since last relapse, disease duration, number of prior relapses, expanded disability status scale (EDSS) score, and number of gadolinium-enhanced lesions. We also developed a web application that calculates an individual's probability of relapsing within the next 2 years. The optimism-corrected c-statistic is 0.65 and the optimism-corrected calibration slope is 0.92. For threshold probabilities between 15 and 30%, the "treat based on the prognostic model" strategy leads to the highest net benefit and hence is considered the most clinically useful strategy. CONCLUSIONS The prognostic model we developed offers several advantages in comparison to previously published prognostic models on RRMS. Importantly, we assessed the potential clinical benefit to better quantify the clinical impact of the model. Our web application, once externally validated in the future, could be used by patients and doctors to calculate the individualized probability of relapsing within 2 years and to inform the management of their disease

Explainable Artificial Intelligence to predict clinical outcomes in type 1 diabetes and relapsing-remitting multiple sclerosis adult patients

Artificial intelligence (AI) is increasingly being used to improve patient care and management. In this paper, we propose explainable AI (XAI) models for predicting severe hypoglycemia (SH) and diabetic ketoacidosis (DKA) episodes in adults with type 1 diabetes (T1D) and relapses in adults with relapsing-remitting multiple sclerosis (RRMS). We follow a three-step process in this study: (1) develop baseline machine learning (ML) models, (2) improve the models using ReliefF feature selection technique, and develop sex-stratified models, (3) explain the models and their results using SHapley Additive exPlanations (SHAP). We built six ML models (XGBoost, LightGBM, CatBoost, AdaBoost, random forest, and linear regression) for all scenarios. Applying the ReliefF feature selection led to improved model performance in predicting all outcomes compared to the baseline models. Additionally, sex-stratified models further improved the prediction of SH episodes and relapses. The F1 scores for predicting SH episodes in male and female patients were 84.07% and 84.95%, respectively, and the DKA prediction model achieved an F1 score of 78.67%. The proposed relapse prediction models outperformed existing models with F1 scores of 84.55% (males) and 76.11% (females), and ROCs of 70.26% (males) and 69.05% (females). Our results highlight the importance of considering sex differences, socioeconomic factors, and physical and mental health in medical outcome prediction. Boosting ML algorithms were found to be effective in detecting SH and DKA in T1D patients and relapses in RRMS patients compared to conventional tree-based ML and statistical models

Explainable Artificial Intelligence to predict clinical outcomes in type 1 diabetes and relapsing-remitting multiple sclerosis adult patients

Abstract Artificial intelligence (AI) is increasingly being used to improve patient care and management. In this paper, we propose explainable AI (XAI) models for predicting severe hypoglycemia (SH) and diabetic ketoacidosis (DKA) episodes in adults with type 1 diabetes (T1D) and relapses in adults with relapsing-remitting multiple sclerosis (RRMS). We follow a three-step process in this study: (1) develop baseline machine learning (ML) models, (2) improve the models using ReliefF feature selection technique, and develop sex-stratified models, (3) explain the models and their results using SHapley Additive exPlanations (SHAP). We built six ML models (XGBoost, LightGBM, CatBoost, AdaBoost, random forest, and linear regression) for all scenarios. Applying the ReliefF feature selection led to improved model performance in predicting all outcomes compared to the baseline models. Additionally, sex-stratified models further improved the prediction of SH episodes and relapses. The F1 scores for predicting SH episodes in male and female patients were 84.07% and 84.95%, respectively, and the DKA prediction model achieved an F1 score of 78.67%. The proposed relapse prediction models outperformed existing models with F1 scores of 84.55% (males) and 76.11% (females), and ROCs of 70.26% (males) and 69.05% (females). Our results highlight the importance of considering sex differences, socioeconomic factors, and physical and mental health in medical outcome prediction. Boosting ML algorithms were found to be effective in detecting SH and DKA in T1D patients and relapses in RRMS patients compared to conventional tree-based ML and statistical models